NII , Japan at the first THUMOS Workshop 2013

We submit the results of our system to the THUMOS’13 Challenge on Recognition Task. We apply a multi-modal approach to recognize actions on UCF101 dataset. These features include Dense Trajectories (motion features), SIFT (image features) and MFCC (audio features). Moreover, we also use the Fisher vector encoding, which is a state-of-theart feature representation on popular image classification and action recognition datasets. 1. Classification Framework The action classification framework is shown in Figure 1. Basically, it consists of following steps: feature extraction, feature encoding and action classification. 1.1. Feature Extraction We use feature from different modalities to model action in UCF101: motion features, still image features, and audio features. We use the Dense Trajectories [7] with Motion Boundary Histogram (MBH) descriptor for capturing moving patterns in video. This is a state of the art motion feature that is suitable for realistic videos because it can suppress camera motions. We use the standard SIFT [4] with Hessian Laplace detector for extracting still image feature. Finally, the standard MFCC [3] feature is used to capture audio features. Audio segment unit is set to 25 ms and the overlapping parameter is set to 10 ms. The 13d MFCCs along with each first and second derivatives are used for representing each audio segment. 1.2. Fisher Vector Encoding Fisher Vector [5] representation is a newly developed feature encoding technique where the mean and variance of local descriptors that belong to each cluster are also considered. Therefore, Fisher vector encodes more information than the bag-of-words feature encoding. Following the standard implementation of Fisher vector, we use the codebook size of 256 clusters which are generated using the Gaussian Mixture Model (GMM). We further improve the expressiveness of Fisher vector by applying PCA for reducing feature dimension, i.e from 128d to 80d for SIFT and from 192d to 128d for MBH descriptor. 1.3. Action Classification We use the popular Support Vector Machine (SVM) for classification. All the positive videos are considered as positive samples and the remaining videos are considered as negative samples. LibSVM [1] is incorporated to our system for learning steps because it is a standard implementation for SVM. We use linear kernel for training features encoded by Fisher vector. Furthermore, we also utilized the pre-computed kernel technique to reduce the training time. This technique is especially useful when the number of action classes are large.